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ABSTRACT 

Outcomes of guidance can be assessed at these three levels: 
individual, where they are immediate; organizational, where they are 
intermediate; or societal, where they are ultimate. They may result in 
learning outcomes, school effectiveness, economic benefits, or social 
benefits. Various designs of outcome evaluation are described and 
recommendations are made as to which design is appropriate based upon the 
objectives of policy and the type of evidence available. A review of 
literature finds that convincing evidence of positive outcomes of career 
guidance has been difficult to quantify. Future, research must be more 
longitudinal in order to identify long-term, deep-seated effects and that the 
three key players in the establishment of criteria for measurement of 
outcomes policy makers, practitioners, and researchers must engage in 
dialogue and agree on common approaches. (As part of a study of guidance 
systems in European Union member states, this paper considers different 
approaches to measurement of outcomes from career information and guidance 
service delivery, provides exemplars of different approaches, summarizes 
reviews of evidence using these approaches, and outlines policy priorities 
for the collection of evidence in the future and for effective sharing and 
dissemination of such evidence. It includes 86 references and 1 chart.) (MO) 
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As part of a study of guidance systems in Member State, in a perspective of lifelong 
learning, which the European Commission is undertaking in co-operation with OECD, the 
National Institute for Careers Education and Counselling (NICEC) was commissioned to 
prepare a paper on “evaluating outcomes in guidance service delivery”. The aim was to 
consider different approaches to measurement of outcomes from career information and 
guidance service delivery, provide exemplars of different approaches, summarise reviews of 
available evidence using these approaches, and outline policy priorities for the collection of 
such evidence in future, and for effective sharing and dissemination of such evidence. 

It is important, at the outset, to recognise the policy context in which career information 
and guidance service delivery is located. While it is true that many recipients of services, and 
most certainly career guidance practitioners are convinced of the value of these interventions, 
policy makers require evidence of this in order to justify the outlay of significant tracts of 
government funding. As Watts (1999: 13) suggests, a distinction may be made between 
career guidance being viewed as a “worthy private good”, which bestows benefits to 
individuals who “should have a civic right to have access to it regardless of the resources at 
their private disposal, and as a "'public good” which generates social and economic benefits 
over and above those accruing to the individuals who receive it. 

The paper draws extensively on the earlier work of one of the authors, John Killeen, as 
well as Watts (1999) and a more recent review undertaken in the UK (Hughes et al, 2002). 
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BACKGROUND ISSUES 



It is important to recognise that attempts to undertake definitive studies, or provide 
conclusive evidence of the outcomes of career guidance activities continue to be beset by a 
recurring set of issues, which have been summarised by Hughes et al (2002) as follows: 

• there are a wide range of factors which influence individual career choice and 
decision-making, and/or which can impact on outcomes; 

• career guidance is frequently not a discrete input, but rather is embedded in other 
contexts, such as learning provision, employer/employee relationships, and or 
within multi-strand initiatives; 

• comparing the evidence available in different studies is problematic when the nature 
of career guidance, the depth of work undertaken and client groups, vary 
considerably; 

• there is not an agreed set of outcome measures for career guidance, or common 
methods of collecting output, or outcome data, except in the case of a limited 
number of discrete programmes/areas of work”. 



(Hughes et al, 2002:19) 

Definitional problems also abound when addressing this area. The first of these may be 
in attempting to define what guidance is, and what it is intended to do. In seeking to identify 
and measure the outcomes of career guidance, a clearly delineated ‘guidance’ intervention is 
desirable, although not always achievable. Moreover, it can be argued that effective career 
guidance is a process which, over time, is made up of a number of individual interventions, 
possibly of different types. 

Allied to this is the question of what should be included in any study as constituting a 
career guidance intervention. A wide range of activities, from telephone call services to in- 
depth face-to-face interviews may be included here. In practical terms, it may be advisable to 
adopt the approach taken by Killeen and White (2000) in their study of the impact of career 
guidance on adult employed people. They chose to restrict their sample to recipients of 
guidance services which were publicly available, across a range of mechanisms. 

Similar concerns arise when considering what constitutes a positive effect. For example, 
while reduced student drop-out from courses may be deemed to be a positive effect of the 
input of information, advice and guidance, it should be remembered that ‘drop-out’ can also 
be positive, as when it is the result of finding a job or of transferring to a more appropriate or 
higher level course. 

In order to ensure a clarity of focus in the paper, and to utilise the available resources 
most effectively, ‘approaches to measurement of outcomes’, as mentioned in the 
specification, will be interpreted as ‘the general types of study designs through which 
outcomes are estimated’ and not as ‘approaches to operationalisation and measurement of the 
outcome variables’. 
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Another major issue which must be confronted when assessing the degree of reliability 
to place on measures of the impact of guidance is in differentiating the effect of the guidance 
from that of a whole series of other contributory factors. As Herr (2001) states; “the frequent 
difficulty in research and, by extrapolation, cost/benefit analyses in career development lies in 
verifying the relationship between quality outcomes and the specific career development 
interventions, often the measures of such outcomes are crudely measured and the career 
development interventions are not adequately isolated or the ‘treatment’ described”. 

Underlying some of the above statements may be a concern that the search for concrete, 
readily measurable outcomes is a flawed exercise, as it may involve according dubious 
quantitative specificity to what are essentially qualitative and sometimes highly subjective 
factors. For example, Herr (2001) quotes a response from a German contributor: “who needs 
proof of something as obvious as these fundamental truths about mankind and its typical 
behaviour? Forget about ‘proof anyway: proof is a fictitious concept of the fading 
mechanical era”. While such sentiments may find favour among those who have a deep- 
seated belief in the efficacy of career guidance, it is unlikely satisfy policy makers who are 
required to be accountable for investing public funds. 



TYPES OF OUTCOMES FROM CAREER INFORMATION AND GUIDANCE 

DELIVERY 



Watts (1999) identifies three locations for assessing the outcomes of guidance, each of 
which has a different timescale attached to it. These are: the individual, where the outcomes, 
which are designated as ‘learning outcomes’ are immediate; organisational, where the 
outcomes, which are designated as ‘school effectiveness (e.g.)’ are intermediate; and societal, 
where the outcomes, which are designated as ‘economic benefits and social benefits’ are 
ultimate. The time period over which outcome measures are applied is extremely important, 
as it could be argued that longer-term outcomes, for the individual, the organisation, and 
society as a whole, are likely to reflect the true benefits of guidance. However, what may be 
deemed appropriate or desirable timescales from an evaluation perspective run the risk of 
being beyond the scope and budget of individual research projects, and of offering 
insufficient immediately relevant data for policy makers. 

From this, he focuses on the four types of outcomes, with economic benefits and social 
benefits being treated separately. As far as ‘learning outcomes’ are concerned. Watts posits 
that these are likely to continue to provide the most readily usable measures, because of their 
appropriateness, in that they reflect the guidance practitioner’s concern to offer a range of 
alternatives, rather than being overly prescriptive, and practicability, in that the immediate, or 
short-term effects are able to be measured with relative ease. 

Within the category of organisations for which benefits may be derived from guidance. 
Watts includes employers, as well as schools and other education and training providers. For 
the latter, the positive outcomes will be greater efficiency resulting from learners being 
engaged on more appropriate course provision, and therefore reducing drop-out and 
enhancing attainment. Retention, productivity and greater efficiency and effectiveness among 
employees will accrue to employers from suitable guidance activity. 

There is a danger that in seeking to identify and measure outcomes from career guidance 
activity, attempts are made to incorporate too wide a spectrum of effects, some of which may 
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go far beyond what might be regarded as the principle focus and remit of that activity. Watts 
refers to these as side-effects, which may include aspects such as “academic motivation and 
attainment” and other attitudinal changes. However, it is difficult to see how studies of the 
impact of career guidance activity can be meaningful if they do not allow for what might 
constitute life-changing effects of interventions, which may or may not be readily apparent in 
terms of easily observable or tangible outcomes, such as the take-up of learning opportunities, 
or entry into employment. 

Certainly the ability of career guidance to initiate or encourage attitudinal changes is 
held to be highly significant in the literature. Hughes et al (2002:10), in considering the 
economic benefits of guidance, provide a set of categories for measuring this impact, with 
“motivation and attitudinal change” being placed alongside learning outcomes; participation 
in learning; student retention and achievement; job search/reduced unemployment; 
employment; and economy. Moreover, they emphasise the fact that Killeen et al (1992) 
“explicitly state that attitudinal change and learning outcomes are important ‘precursors’ to 
the economic benefits of guidance”. Earlier, within a classification of learning outcomes, 
Killeen and Kidd (1991) placed “attitudes” alongside decision-making skills; self-awareness; 
opportunity awareness; certainty of preference; and transition skills. 

In terms of economic benefits, these have been characterised as comprising three 
elements: individual decisions; market failures; and institutional reforms (Watts, 1999:15). 
As far as individual decisions are concerned, effective guidance can ensure that job search 
and employment-related decisions are better informed, thereby resulting in a more efficient 
workforce and greater complementarity between the supply of, and demand for labour. 
Similarly, decisions relating to learning opportunities are more likely to be appropriate and 
lead to ‘successful’ outcomes. Herr (2001) characterises the economic benefits derived from 
career guidance by individuals as being “their ability to secure jobs with improved pay, in 
shortened periods of unemployment, in obtaining greater congruence between personal 
interests and abilities in a job chosen and in the experience of extended tenure in that job”. 

The chances of market failure can be alleviated by effective career guidance, through 
reducing the propensity of learners to embark upon and subsequently drop out of education or 
training courses, by reducing the amount of mismatch between job vacancies and the 
available pool of unemployed labour, and by re-enervating previously discouraged workers, 
who were not aware of potential opportunities for them to regain employment. A reduction in 
the length of job search may also be a valid ‘positive’ measure. 

Effective career guidance can assist institutional reforms by ensuring that potential 
participants are aware of any developments which may impact on their decisions, such as 
choice of course or institution. 

Turning to the social benefits of career guidance. Watts contends that “the case for 
guidance having a role in reducing social exclusion is not difficult to make”, in that career 
guidance focuses on encouraging participation in learning and in employment. It can 
therefore be seen as a force for preventing ‘at risk’ individuals from becoming socially 
excluded, and alleviating the situation of those who have become excluded, by assisting them 
to be re-engaged through education, training or employment. 

It is important to emphasise here the difficulties of defining social benefits, for while 
there are clearly benefits to be derived by society (and the economy) as a whole from 
effective career guidance, for example through attitudinal shifts which result in a greater 
attachment to prevalent societal values, there are also what may be termed social benefits 
which accrue to individuals. These would include an improved ‘quality of life’ (Watts, 
1999:17). Moreover, in both cases, great difficulties arise in identifying, differentiating and, 
crucially, in measuring those benefits. Herr (2001) contends that “for society at large, a 
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purposeful, productive, teachable work force may enhance the competitive edge of a nation in 
competing for market share in a global economy, in being innovative, and in maximizing high 
skill/high pay jobs in the occupational structure”. While it may be difficult to argue against 
this, there remains the great difficulty of establishing the part played by, and quantifying the 
contribution of, career guidance to the creation of such a workforce. 

Mayston (2002a) contends that “there are a number of important wider social benefits 
which are likely to be generated by high quality career guidance, and which could be included 
in a cost-benefit analysis of such career guidance” (p.20). These are indeed “wider” benefits, 
for they comprise: increased tax yields to the Exchequer; reductions in unemployment and 
other social security costs to the Exchequer; reductions in health care costs on the National 
Health Service; reductions in the frequency and costs of crime; macro-economic benefits; and 
net social benefits. 

In discussing ‘quality outcomes’ from career guidance activity, Savickas (2001), on the 
basis of submissions from correspondents from different countries around the world, 
concluded that, with the exception of Spain, “policy makers and practitioners view quality 
outcomes from two distinct vantage points. Policy makers view outcomes from an objective 
perspective that focuses on workers’ success, satisfactoriness and adjustment. In contrast, 
practitioners view outcomes from a subjective perspective that focuses on worker’s job 
satisfaction, personal development, and maturity. Policy makers look at what workers 
contribute to the economy whereas practitioners look at what workers receive from 
employment”. He then goes on to differentiate the two perspectives as representing 
quantitative quality outcomes (policy makers) and qualitative quality outcomes 
(practitioners). This is an interesting stance which, while being over-simplistic, does point up 
the difference between what policy makers and practitioners may regard as appropriate 
measures of successful outcomes of career guidance. 

Herr (2001) addresses this issue of diverging interests between policy makers and 
practitioners from a different perspective. He contends that whereas policy makers are 
concerned with macro-economic issues, such as overall levels of employment and 
unemployment, labour productivity, the position of marginalised or disadvantaged groups, the 
effective use of resources and “in some cases, social justice”, practitioners are perceived to 
have a focus at the “micro-level”, with a concern for individual clients and how they may 
achieve learning or employment goals which most nearly match their aspirations or provide 
job satisfaction. However, in line with Savickas’ assertions, he queries whether reliable 
quantitative indicators can be developed to provide a measure of factors such as 
“independence/self-directedness, self-reliance “, which are “subjective and qualitative in their 
essence”. 



CRITERIA FOR OUTCOMES 



As the titles of some of the key references suggest, an ongoing focus of attention when 
assessing the outcomes of career guidance has been the costs and benefits of that activity (eg 
Hughes et al, 2002; Herr, 2001; Mayston, 2002; Watts, 1999). However, in his synthesis of a 
range of country papers, Herr (2001) posits that “the implementation of costs/benefits 
analyses as a national strategy has not become an empirical process, but rather a presumptive 
process; presumptive in the sense that benefits are expected to flow from the implementation 
of career services, although such hypotheses have not been tested”. This again highlights the 
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difficulties inherent in identifying exactly what benefits may be anticipated from career 
guidance, and points up the need for developing an understanding of these benefits which is 
common not just across countries, but between policy makers, practitioners and researchers. 

A key issue for Herr in the development of such an understanding is the lack of 
agreement on “the language of career development initiatives”. Certainly, the development of 
common, agreed terminology for different activities may assist in enabling evaluation or 
measurement of initiatives to be generalisable and comparable, and thereby lead to the 
creation of a more substantial body of evidence on which to base arguments about the 
reliability of ‘outcomes’ emanating from career guidance. 

A New Zealand based study of outcomes from career intervention conducted in 1999 
identified ‘both ‘soft’ outcomes, related to how respondents felt about their career goals, and 
‘hard’ outcomes, which were measures of what they did to achieve those goals. The 
definition of career intervention used was “having attended a career guidance session with a 
career counsellor”. 

As far as cost/benefit analysis is concerned, Herr (2001) poses the question: “is it 
possible to implement comprehensive cost/benefit analyses without a research base that 
addresses the effects of various career development interventions on different populations 
with different career goals?”. 



DESIGNS OF OUTCOME EVALUATION 



In the evaluation of state interventions into the labour and human capital markets, terms 
are increasingly used which, so far as we are concerned with career information and guidance, 
may be represented as follows. 

Treatment. An information or guidance technique, process, programme, practitioner or 
service, the effect of which is to be estimated. 

Local average treatment effect (LATE). (Imbens and Angrist, 1994). The average effect 
(or, estimated average effect), of guidance treatment on the treated in an observed sub- 
population of those eligible for treatment (or, in a sub-population of which the observed, 
treated subjects are an unbiased sample). It is sometimes remarked that the samples in studies 
of guidance effectiveness may not be ‘representative’ and that there are doubts about how, or 
how far, results should be ‘generalised’. Providing samples are taken from a defined 
population without bias, or, so as to allows correction of bias, then it is possible to define a 
LATE. 

Average treatment effect on the treated (TT). Mean effect of treatment as currently 
allocated, or, on treated subjects in the whole eligible population. 

Average treatment effect on the untreated (TU). Estimated mean effect of treatment on 
all untreated subjects in the whole population eligible for treatment. That is to say, the 
estimated average treatment effect on the untreated, should they have been treated. Since 
treated and untreated may be unlike, so may TU differ from TT. 
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Average treatment effect (ATE), Estimated mean effect of treatment on all treated and 
untreated individuals within the population eligible for treatment. 

Since selection to treatment within the eligible guidance population is usually non- 
random, it usually is not possible to assume that:- 



TT = TU = ATE 



Since studies of guidance often consider sub-populations of those eligible for treatment, 
it often is not possible to assume that:- 

LATE = ATE 

And since guidance samples are often idiosyncratic and not taken from a defined 
population in a way which is either unbiased, or, allows correction of bias, LATE is 
frequently undefined. 

Why are these so? For many treatments, such as training, it is not necessary to make the 
general case. That is to say, it is already known, or assumed beyond question, (a) that the 
treatment can lead to learning outcomes (knowledge, skills, work habits, etc.) which it has as 
an objective to develop, (b) that these learning outcomes are in demand in the labour market 
and (c) that some treated subjects will supply this demand. In consequence, the aim of 
econometric evaluation of such treatments is, ultimately, to allow cost-benefit analysis. In 
this circumstance, the main policy questions are as follows: 

• what are the existing private and public rates of return on investment in treatment? 

• do these rates vary according to type of treatment and/or according type of treated 
subject? 

• what would be the implications for these things of inducing a higher (or a lower) 
rate of treatment participation, or, of altering eligibility criteria, or, of substituting 
treatments for one another, and so on? 

However, some treatments exist at the ends of relatively lengthy chains of connection to 
a priori assumptions relating to strategic policy goals, for example, the a priori assumption 
that the labour and human capital markets suffer serious imperfections of information which 
must be remedied, or, that there is a need to persuade adults of the advantages of continuing 
education and training (GET), or welfare-dependent people of the advantages of employment. 
Others exist because of the existence of a social problem (e.g. unemployment), which cannot 
be solved by fiat, but where ‘no treatment’ is not a viable policy option within a democratic 
system. In other cases an organisational problem may exist (e.g. failure of students to make 
realistic or stable option choices at the point of entry to or within schools and colleges) for 
which non-coercive intervention into decision processes appears to be the only tenable 
remedy. From the perspective of policy (although not from all perspectives, including some 
adopted by guidance practitioners), ‘career information and guidance’ describes, in indefinite 
fashion, what is done (the treatments) corresponding to these and similar circumstances and 
objectives. In each case, questions of the following kinds are asked. ‘Does a treatment 
produce effects envisaged in policy?’ ‘Can a treatment be found which is capable of 
producing effects envisaged in policy?’ ‘Can evidence be found that one treatment is more 
effective than another in producing effects envisaged in policy?’. 

In short, the questions addressed in econometric evaluation are seldom addressed in 
guidance evaluation, because two other, much more fundamental, tasks remain paramount. 
The first of these is to test the null hypothesis (to establish that there are effects), and the 
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second is to test for significant difference between the effect sizes of alternative treatments. 
Fortuitously, the academic discipline most influential over both the practice and the most 
rigorous forms of evaluation of guidance has been psychology and these are the kinds of tasks 
which it conventionally adopts, in a somewhat ‘clinical’ approach to evaluation. 

The methods used in studies of the effectiveness of guidance could, in principle, be 
ranked in approximate order, from those which do not test the hypothesis of an effect on the 
outcome, to those which give unbiased estimates of the ATE and/or unbiased estimates of one 
or more LATE relevant to policy. However, in order to review existing studies, it is 
necessary to concentrate on the ability of designs to demonstrate gains in the samples studied 
(internal validity), consistent with the approach actually taken, and not include the ability of 
studies to estimate ATE/LATE. Most studies can also be assessed on this dimension. The 
implication of what has already been said is that, in the great majority of cases, the rank of 
actual studied is low. No attempt is made here to catalogue all possible designs, since, for 
example, there is a very large family of experimental designs, most of which have never been 
used in studies of guidance, or are very seldom used. 

Of course, ultimate policy goals may be stated in terms of GNP, net exchequer saving, 
and so forth, but studies of guidance do not actually work at this level. For the present 
purpose, it is sufficient to assume that policy goals are formulated in relation to individuals or 
small aggregates, such as families, schools, employing organisations or small administrative 
districts. 

Irrespective of the unit of analysis, and because serious national estimates are not 
attempted, estimates of the outcomes of guidance lack adjustment for displacement. That is 
to say, when guidance is evaluated, it is not asked if a restricted set of outcomes (e.g. 
employment opportunities) is redistributed by means of treatment. Sometimes redistribution 
is a plausible objective of policy on the ground of equity. For example, guidance may be 
given to a group which suffers employment discrimination. In this case, however, the aim is 
to redistribute employment from those not eligible for treatment, to those who are eligible for 
treatment. It is not the aim of policy to redistribute employment from those eligible but 
untreated, to those eligible and treated. Sometimes, the aims associated with guidance, such 
as improved fit between the jobs accessed and the aspirations of the job-seekers, imply 
redistribution with net gain, as in job satisfaction. For example, assume that, over some 
period, amongst those eligible, the treated have a higher rate of employment or job 
satisfaction than the untreated. At one extreme, all of the extra employment or job satisfaction 
of treated subjects equals an increase in the number of people in employment or the total 
amount of job satisfaction. But at the other extreme, employment or job satisfaction is 
thought a zero-sum game and guidance treatments confer competitive advantage equal to the 
guidance effect. Now, assuming that the outcome is available only to those eligible for 
treatment, this gives: 



TT + TU = 0 = ATE 

Of course, assuming a guidance treatment effect on the treated, a much more realistic 
assumption is that the rate of displacement falls between these poles, but is unlikely to lie 
precisely at one or the other. It may, therefore, seem odd that such estimates are not insisted 
upon by policy makers. But there is a simple explanation. The outcomes which are most 
easily demonstrated do not suffer from a credible threat of this kind. For example, the 
learning outcomes of guidance are very difficult to conceptualise as competitively distributed. 
On the other hand, the outcomes most susceptible to this threat are far from adequately 
demonstrated. Thus, at the present state of development of guidance evaluation, the 
estimation of displacement effects may be regarded as a luxury. 
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Key issues to be considered when assessing the efficacy of methodological approaches 
to measuring the outcomes of career guidance activity include the restrictions which are 
placed on the timescale, and the difficulty of incorporating control groups or an appropriate 
counterfactual. In terms of the former, while it can justifiably be argued that the substantive 
benefits accruing from effective career guidance can take many years to emerge, it is also true 
that the longer the time span, the more difficult it becomes to differentiate the impact of the 
guidance intervention from an cumulatively increasing number of other potential factors. 

From a methodological standpoint, randomised trials may be the most suitable approach 
to be adopted, offering the prospect of robust data on which to base conclusions and 
recommendations. In a standard experimental situation, people would be randomly assigned 
to an experimental group or a control group, and career guidance administered to those in the 
experimental group while being withheld from those in the control group. The control group 
provides the counterfactual and elicits data on what happens in the absence of the treatment. 
The difference in the outcomes scores would then represent the impact of the guidance 
intervention. 

However, this method may be difficult to apply in practice, either because of threats to 
‘internal validity’ (to causal inference), or because of reluctance to extend the use of 
randomisation to labour market interventions or into educational settings. In principle, there 
is scope for more use of the method to study short-run guidance effects, as occurs in the US. 
But, when randomised trials are not possible, the main options available have, until recently, 
been ‘regression adjustment* - the exertion of statistical control and adjustment - very simple 
sample matching, or a fairly crude combination of these two things. However, each poses 
many problems and, following groundwork by statisticians (notably by Rubin and by 
Heckman) a new option, better-suited to the demands of programme evaluation, is now 
available. The utility of this method in guidance evaluation has already been shown by 
Killeen and White (2000), who found that by using the propensity matched score method, 
estimates of the impacts on outcomes tended to be less strong than was the case with 
regression based analysis. 

The essential elements of this approach may be summarised as follows. Basic 
assumptions are or imply (a) that control sample members could have received the ‘treatment’ 
(in this case, guidance) and (b) that the factors disposing individuals to be treated are 
observable. The method is a development of simple ‘matching on observables’, in which 
untreated sample members are matched to their treated counterparts. However, in simple 
matching on observables, and with interaction, there is geometric increase in the number of 
cells to be matched. Thus, it is usually possible to match only on a handful of characteristics 
such as age group, sex and so on, leading to the need for further adjustment, normally in 
violation of sampling assumptions. However, it has been shown that a result equivalent to 
matching on the set of observables upon which treatment is conditional may be obtained by 
matching, instead, on the probability of participation in the treatment (the ‘propensity score’). 
A propensity score is obtained for every case (both treated and untreated) by modelling 
participation. As treatment is usually binary (it happens or it does not), the method is usually 
simple logit or probit. The combined treatment sample and sample of potential control 
subjects are used to obtain this model. The propensity score of each case is then computed 
from the participation model. Following this, control cases are selected from the pool of 
potential control subjects according to their match to treated cases. Sampling may be with 
replacement and more than one control case may be selected for each treated case, thus 
allowing a number of types of weighted comparison sample to be created. In effect, the aim 
is to construct an approximation to a randomly-assigned control sample. Thereafter, simple, 
easily-communicated comparisons may be made between samples and the outcomes 
estimated. 
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Unlike regression adjustment, which should not, but repeatedly does, distract us from 
this issue, propensity score matching reminds us of the importance of an adequate 
appreciation of the factors which lead adults to use guidance and of the extent to which this is 
and is not a random process. By combining qualitative with quantitative research, it should 
be possible to get a better understanding of these factors than hitherto. Guidance 
effectiveness research is as good as the counterfactual and this knowledge, coupled with an 
appreciation of alternative determinants of outcomes, is vital to refinement of the 
counterfactual in any non-experimental approach. This should be cumulative across studies, 
although we would suggest that the currently most apt method by which to test the 
counterfactual is the one just described. 

Similarly to designs, outputs and outcomes themselves may be ranked according to their 
proximity to the objectives of policy, on the assumption that outputs and outcomes which are 
plausible antecedents of policy goals are of less importance than outcomes which actually 
represent those goals. As we have already indicated, the latter remain the most problematic. 
Taken together, the rankings of designs and outcomes form a matrix which may be read from 
top left {prima facie but unconvincing evidence of effects which may be antecedents of 
outcomes central to policy) to bottom right (convincing, demonstrations of effects on 
outcomes which represent policy goals, but from which unbiased estimates of ATE and/or 
LATE are not usually possible). The further we move in this direction, the less evidence is 
available. Thus some of our recommendations are incipient in the figure below. 
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DISCUSSION OF EVIDENCE 



The difficulties of producing convincing evidence of the positive outcomes of career guidance can be 
gauged from the conclusion reached by a review of international literature (ACNielsen, 1999), which 
stated that “although there is little hard evidence to back up the proposition that career guidance has 
economic and social benefits, the rationale behind such a proposition is sound”. While this represents an 
accurate appraisal of the position and few involved in career guidance activity would disagree with the 
“proposition”, it falls some way short of the economically solid case which is likely to be needed to 
persuade government departments which control the purse strings, to provide substantial investment in 
career guidance. 

In the New Zealand study referred to earlier (ACNielsen, 1999), both qualitative and quantitative 
interview techniques were used, and a sample of 400 respondents were followed up for up to a year after 
their experience of the guidance intervention. Interestingly, in contrast to the somewhat downbeat tone of 
the conclusion to the literature review, the New Zealand study found that eighty per cent of the respondents 
claimed that they had made changes related to work as a result of their career counselling session. Of 
these, 28 per cent had entered a job which matched their aspirations or skills and 27 per cent had embarked 
on relevant education or training. Moreover, 86 per cent of the sample felt that Careers Services had been 
influential in their employment-related decision-making. 

As far as ‘soff outcomes were concerned, 54 per cent felt that Careers Services had provided them 
with a range of options for future employment and career direction, while 36 per cent suggested that the 
input from Careers Services had enhanced their self-confidence and self-esteem, especially where job- 
seeking was concerned. 

As was alluded to earlier, altitudinal change and shifts in individual motivation are considered to be 
important outcomes deriving from career guidance interventions. Hughes et al (2002) provide the 
following examples, largely drawn from evaluations of programmes or initiatives which incorporated some 
element of career guidance, showing a positive outcome emanating from the intervention: “ 

“Bysshe and Parsons (1999) reported changes in self-confidence in ‘callers’ to Learning Direct 

James (2001) found that patients who consulted a ‘Learning Adviser’ based within health centre 
settings reported that their motivation and self-esteem had improved. 

Morris et al (1999) highlighted that access to good quality careers guidance was a critical key 
factor in raising young people’s levels of awareness and positive attitudes towards vocational 
training. 

Barham, Hughes and Morgan (2000) in their study of New Start pilots found that some young 
people were thought to have made significant gains in terms of self-confidence and self- 
esteem. 

Hasluck (2000a) found that young people (18-24 year olds) on New Deal reported improved 
motivation and self-confidence. 
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Winterbotham et al (2001) reported on the impact and effectiveness of the New Deal 25+ for 
long-term unemployed. They indicated that those who had participated in courses in job 
search skills as part of the ‘Gateway’ had gained confidence. 

Coopers and Lybrand (1995) evaluated the ‘Skill Choice’ programme for adults and concluded 
that participants had improved their attitude towards training, becoming more qualified, and 
their own career development. 

Skills Task Force (2000) Skills for All: Research Report from the National Skills Task Force. 
DFEE, Sheffield: 

‘A significant proportion of adults have no interest in further learning. This lack of demand is 
likely to act as a barrier to future development for people with low levels of skill as much as it is 
for those lacking basic skills. Pressures from work and family were quoted by one in five of those 
reported in NALS as reasons for not participating in learning. However, a lack of confidence 
related to age and ability, and ignorance about opportunities were quoted by similar proportions. 
This suggests that better guidance or counselling could persuade some of these adults about the 
merits of further learning’.” 



(Hughes et al, 2002:10) 

While this list provides powerful support for the efficacy of career guidance, it also underlines one of 
the great dilemmas when attempting to make definitive statements about the impact of that guidance: that 
is in being able to differentiate the impact of the ‘guidance’ component from other aspects. In this way, it 
may point up the fact that guidance can be highly influential in conjunction with other types of 
intervention, such as the provision of training. It does, however, remind us of the difficulties of making 
claims for the impact of career guidance per se. 

As far as learning outcomes are concerned. Watts (1999:14) asserts that “evidence on the learning 
outcomes of guidance is substantial and convincing”. This statement is based on an appraisal of forty 
studies, which overwhelmingly pointed to some positive outcomes, and is supported by American studies 
of a similar nature (Spokane and Oliver, 1983; Oliver and Spokane, 1988). Overall, however, Watts is at 
pains to advise against placing undue reliance on what is admittedly limited evidence, which is partly 
attributable to the acknowledged difficulties of undertaking studies which have sufficiently large sample 
sizes and appropriately sensitive data-gathering techniques and instruments. Indeed, Killeen et al (1992) 
found no evidence of guidance interventions leading to increased participation in learning. 

Here, the problem of disentangling any specific ‘guidance’ effect from a wider range of contributory 
factors is again evident, and Hughes et al (2002) point to a number of studies which cite guidance as 
having contributed to an identifiable increase in participation (see Killeen, 1996; MORI, 1996; Coopers 
and Lybrand, 1995; Killeen and White, 2000; and MORI, 2001. This leads them to conclude that “there is 
now reasonably strong UK quasi-experimental evidence that voluntary exposure to guidance increases the 
probability of adult participation in continuing education and training, relative to similar individuals not 
exposed to guidance” (p. 12). 

Killeen and Kidd’s (1991) earlier study asserted that, for each of the categories of learning outcomes 
they identified (i.e. attitudes; decision-making skills; self-awareness; opportunity of awareness; certainty of 
preference; transition skills), their review, which was largely based on American data, indicated that 
positive impacts could be identified. Hughes et al (2002) confirm this finding, citing a range of more 
recent studies predominantly from the UK (see Bysshe and Parsons, 1999; James, 2001; Brooks, 1998; 
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Sims et al, 2001; Killeen, 1996a; Hasluck, 2000a; Davies and Irving, 2000; Winterbotham et al, 2001; Van 
Reenen, 2001; MORI, 1996; and Killeen and White, 2000) 

Focusing on the impact of careers programmes in schools in the UK, Andrews et al (1998) suggested 
that the evidence could be viewed from three perspectives: from a positively support perspective, careers 
work was seen to contribute to enhanced achievement, on the part of pupils, in addition to other beneficial 
effects on decision-making; from a neutral perspective, it was regarded as having no effect on performance 
or behaviour; while from a distractive perspective, it was seen to shift the focus of pupils in an unhelpful 
way. This hardly provides compelling support for the efficacy of careers programmes in schools. Nor 
does the summing up by Watts (1999) that “ the most important conclusion to draw from our careful 
review of the research on the effects of careers education and guidance programmes on school 
effectiveness (Killeen, Sammons and Watts, 1999) is that we have found no evidence at all that it has 
negative effects. The research is admittedly limited”. The studies referred to were mainly from the United 
States and pointed to some positive effects on academic achievement (Watts and Herr, 1976; Evans and 
Burck, 1992; Lapan, Gysbers and Sun, 1997). 

Killeen, Sammons and Watts (1999) assert that “there is a strong theoretical case for supposing that 
career education and guidance programmes are likely to have a positive effect on educational motivation 
and hence on academic attainment”. They suggest that such claims are grounded in manifestations of 
motivation theory, including “self-efficacy theory, achievement motivation theory, and attribution theory”. 
However, hard evidence to support these claims is lacking, and reliance is placed on the contention that 
“certainly there is no evidence that careers programmes have negative effects on motivation and 
attainment: the limited evidence we have suggests that they have modest but positive effects on these 
variables”. 

Similarly, the robustness of data which purports to show a causal link between career guidance and 
increases in rates of retention and achievement on education or training courses is debatable. Killeen et al 
(1992) suggested that UK studies which sought to establish this causality were “methodologically flawed, 
inadequately reported, or both”. While more recently it has been re-asserted that inadequate information, 
advice or guidance contributed to student drop-out (eg McGivney, 1996; Sargant, 2000; Morris et al, 
1999), it is once more the case that any increases in student retention and achievement rates may be 
attributable to a series of factors, some of which will be highly specific to a particular course or institution, 
but which could include information, advice and guidance provision. 

Studies providing evidence of the economic benefits of career guidance are thin on the ground. 
Reference has been made to American studies, involving controlled trials, which pointed to career 
guidance having led to individuals entering more suitable jobs, achieving greater job satisfaction and 
experiencing less job turnover than their counterparts who were not in receipt of guidance (see Killeen, 
White and Watts, 1992 for an overview). However, these studies were conducted in the 1920s and 1930s, 
and therefore their relevance may be questionable. More recent (although still dated) studies conducted in 
the United States showed that Job Clubs, which included some career guidance provision in their remit 
were successful in reducing dependency on welfare payments and thereby making considerable cost 
savings. Again, this does not provide conclusive evidence of the likely impact of conventional career 
guidance interventions (Azrin et al, 1980; 1981). 

With reference to UK studies which have addressed similar issues, Hughes et al (2002:14) state that 
“the evidence to date is that quite intensive, multi-method guidance intended to support the job search of 
non- or unemployed people does reduce mean job search time/enhance re-employment rate over the short- 
to-medium term”. 
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Reference was made earlier to the potential benefits of career guidance to employees, notably in 
reducing market failure through enhanced retention and productivity. While there is a dearth of research 
data to substantiate such notions, Hirsh et al (2001) suggest that, from their study, evidence of ‘softer’ 
outcomes accruing to employees, such as motivational and attitudinal shifts could be discerned. 

An interesting dimension to the evidence which is available on take-up and consequent effects of 
career guidance interventions relates to equal opportunities. In the UK, studies have shown that Afro- 
Caribbean and Asian young people are more likely to avail themselves of formal career guidance provision 
than their White counterparts. This is attributed to their relative lack of informed kinship and friendship 
networks (Sillitoe and Meltzer, 1985; Verma and Darby, 1987). 

An example of the way in which the cost of drop-out from education and training courses can be 
measured was a study undertaken in the UK by the Audit Commission and Her Majesty’s Inspectorate 
(1993). Their estimation of the costs of 16-19 year olds failing to achieve their attainment goal stood at 
£500 million. Additional costs, such as those associated with unemployment, would add to this figure. 

To date, most experimental evidence is of US origin and concerns outcomes which can be measured 
immediately after guidance. These studies demonstrate a wide range of positive outcomes, many of which 
may be characterised as ‘learning outcomes of guidance’. However, most US samples are drawn in the 
initial phase of education. Consequently, relatively little is known from this source of relevant effects 
upon adults, even in the short run. 

In the UK, guidance services for adults are strongly-oriented towards continuing education and 
training, but only recently has it been demonstrated that guidance actually raises the probability of 
participation. Killeen and White’s (2000) study demonstrated this both for the short-term unemployed and 
for employed people who seek it on their own behalf It is important to note that, for unemployed people, 
the findings revealed that this was without a corresponding reduction of the chances of re-employment, 
although it was not possible to control for prior motivation or intentions. The same authors have shown 
that guidance has a similar effect on participation in self-funded CET by employed adults. Moreover, it 
was demonstrated that this was net of intentions - use of a guidance service raised the probability of 
participation relative to comparison cases with similar participation intentions. As is common in UK 
evaluation research, data were available only from shortly after initial exposure to guidance and methods 
were non-experi mental. 

Killeen and White’s (2000) study of employed adults was also indicative of an appropriate approach 
to the collection of relevant longitudinal data, within a reasonable timeframe for both policy makers and 
researchers. Postal questionnaires or telephone interviews were administered to a sample of adults two to 
three months after they had experienced some form of guidance intervention. Here, the emphasis was on 
obtaining background information about their personal characteristics, education and training experience, 
occupation, job satisfaction and their exposure to guidance activity. A matched sample of employed 
people who had not received guidance was also interviewed, as the control group. Both groups were then 
followed up some twelve to fifteen months later and information gathered about education and training, job 
satisfaction, earnings and progression. A further follow-up was conducted some eight to ten months later. 
Thus, there was a longitudinal element which enabled the impact of guidance interventions, in terms of 
both learning and employment outcomes and shifts in aspirations and attitudes to be assessed. 
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POLICY IMPLICATIONS 



At a time when a concern for tackling problems arising from increasing numbers of socially excluded 
individuals is a common theme among a wide range of countries, the vital role which effective career 
guidance offers is also generally accepted. This may manifest itself in the targeting of resources, including 
career guidance provision, on those who are ‘at risk’, and in this type of guidance being regarded as a 
component in the drive to inculcate an attachment to notions of learning and to individuals becoming 
proficient in managing their own careers and acquiring and sustaining employability. 

In order to generate more substantive research evidence which is of both immediate and long-term 
relevance to policy makers, Hughes et al (2002:14) make a plea for significant longitudinal studies to be 
undertaken: “... the general case for intensive methods applied to welfare claimants seems reasonably 
secure and it is now time to investigate differential effectiveness by type of guidance, type of client and 
labour market context in order to refine public policy. As the economic arguments concern the economy 
as a whole over the longer term and are not confined only to current welfare savings, there is inadequate 
information concerning the effects of less intensive guidance, of guidance on non-claimant samples, and 
over a somewhat longer time scale”. 

Taking up the point made earlier by Savickas, there is clearly a need for greater discussion, and, 
hopefully, agreement between policy makers, practitioners, and researchers over what will be deemed 
appropriate, desirable and measurable outcomes of career guidance. Commenting on the various country 
papers which formed the basis of his contribution, Savickas states that “of course, once JOINT outcomes 
are identified, we must (as noted by Canada, Ireland and New Zealand) find and agree on criteria for 
evaluating outcomes, not just for the individual but also for societies. There are problems in constructing 
adequate measures. Furthermore, as New Zealand described, there are trade-offs between competing 
objectives such as those focused on long-term sustainable employment and those focused on minimizing 
unemployment by moving people into new jobs as quickly as possible”. Arriving at an agreed 
understanding of the outcomes would seem to be a pre-requisite of a genuine, concerted attempt to develop 
appropriate measures. However, again as Savickas notes, an initial problem is the diffuseness of career 
guidance activity, in a range of settings and with a variety of delivery mechanisms. 

In the same vein, Herr (2001) calls for “definitions of cost-effectiveness or cost/benefit analyses that 
have consistency across national career development initiatives and that can frame such results in terms of 
short-, medium-, and long-term benefits”. 

Irrespective of the approaches taken, a pre-requisite of meaningful outcome evaluation is the 
availability of reliable information. As Mayston (2002b) states: “monitoring the extent to which these 
benefits are achieved, and maximising the value of the careers guidance provided, both depend critically 
upon the effective deployment of detailed information at national and local level by Learning and Skill 
Councils and individual guidance providers”. 

Perhaps one of the most important lessons to be learned from the literature, and one which is alluded 
to by Herr (2001) is that the three key players in the establishment of criteria for, and the measurement of, 
outcomes, namely policy makers, practitioners and researchers appear thus far to have engaged in little 
dialogue with each other, for the purpose of agreeing on common approaches. Without such 
‘rapprochement’, advances in our understanding of the impact of career guidance will continue to be 
piecemeal and restricted to studies which are of necessity constrained by the narrowness of their focus (eg 
on specific guidance interventions) or by the limitations of sample sizes and timeframes. 
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RESEARCH IMPLICATIONS 



On the basis of the foregoing discussion, a number of implications and suggestions for the future 
conduct of research focusing on the outcomes of career guidance delivery can be proposed. 

Firstly, as intimated earlier, the need for research which is longitudinal, and is therefore capable of 
identifying longer-term and possibly more deep-seated effects, not merely in terms of individuals’ 
decision-making, assessment of opportunity structures and behaviour (eg take-up of leaming/job 
opportunities), but also altitudinal shifts and perceptions of self, is of great importance. While 
acknowledging the resource and methodological constraints inherent in undertaking such studies. Watts 
(1999:14) advocates that “if, ... a few strong studies could be mounted of the relationship between 
immediate learning outcomes and the longer-term social and economic outcomes, and if positive 
connections between them could be established, then the learning outcomes could thereafter be regarded 
not only as of value in their own right but also as proxies for the longer-term outcomes (Killeen, White & 
Watts, 1992). To date, such studies have not been conducted”. 

A similar line is taken by Herr (2001), who cites a contribution from Finland: “What is constantly 
needed in the evaluation of the efficiency of services is long-term follow-up studies by which information 
might be obtained on the effects of counseling measures”. 

Allied to this point, assessments and evaluations of specific programmes should seek to take into 
account the impact of the programmes on the aspirations motivation and attitudes to learning of the 
participants. While these may be deemed the responsibility of qualitative investigative methods, as there 
would be a need to identify shifts in ‘softer’ measures, they could easily be accommodated in longitudinal 
studies which were also collecting quantitative data, in terms of respondents’ participation in learning and 
the labour market. 

When designing a research methodology, the complexities of the processes which are being assessed, 
the presence of a multiplicity of factors which, in addition to the career guidance intervention, may be 
affecting the outcomes, and the need to measure the impact on aspects such as self-confidence and self- 
esteem, suggest that extreme caution should be exercised in claiming positive outcomes. Thus, it may be 
that a fundamental premise of credible studies which seek to explore the benefits, especially the social 
benefits of career guidance interventions should be the use of large samples and a range of highly sensitive 
research instruments (Killeen, Watts and Kidd, 1998). 

At the same time, as Watts (1999:18) argues, developing methodologies and research instruments 
which enhance our understanding of the guidance process, rather than seeking to justify, in outcome terms, 
the impact of individual interventions or events, may ultimately be more fruitful for assessing whether 
investment in particular programmes or initiatives is worthwhile. 

One area where more research is urgently needed is examining the sources of information, advice and 
guidance which impact on the decision-making and behaviour of employees. Hawthorn et al (2002), in 
their study of information, advice and guidance for adults in key target groups, contend that “the main and 
overriding reason for adults to access guidance provision is the belief that it may help them to enhance 
their job prospects. This refers both to those who are seeking work, and those in employment who are 
seeking a better job. Within this latter category, employers were found to play a key role in the current 
provision of lAG”. Any research undertaken in this area would need to encompass a whole range of ‘hard’ 
and ‘soft’ outcome measures, and would need to address the outcomes in terms of both the employing 
organisation and the employee. It should be stated that our lack of knowledge about workplace guidance 
processes stems partly from the fact that career-related information, advice and guidance is not widely 
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provided by employers (at least not in a formal sense), and partly because even where this is the case, there 
seems to be an absence of evaluations. 

Among the recommendations made by Hughes et al (2002) are that “there is a need to identify level of 
evidence required to inform public policy debate about appropriate level of guidance investment” (p. 20). 
This is important for researchers and for the design of research studies which seek to shed light on the 
issues of outcomes emanating from guidance activity, for without any clear idea of what will be regarded 
as sufficient evidence of benefits accruing, it is difficult to design and construct studies which fulfil the 
requirements of policy makers (assuming that it is possible to do so). 

Another recommendation is that “short-term evaluation studies should be extended where appropriate, 
to include client research and to enable a longer-term analyses of key findings and trends”. Two points are 
important here. Firstly, it emphasises the need to allow sufficient time for longer-term effects to be 
gauged. Secondly, the focus on ‘client research’ suggests that the views, attitudes, aspirations etc of 
recipients of guidance are crucial to an understanding of the impact of the guidance process, rather than 
relying solely on what appears to be the case from ‘hard’ data, such as take-up, or lack of it, of learning 
opportunities. 




19 



20 



REFERENCES 



ACNielsen (1999) Research into Outcomes from Career Intervention. Career Services: Rapuara, New 
Zealand. 

Allen J., Hansbro J., Mooney P,, (1999) Pathways to Employment: The Final Evaluation ofESF Objective 
3 in Britain (1994-9), DfEE, London 

Allies, S, (1997) Learning for life. Adults Learning Vol 8 (3): 66-67, 

Andrews, D., Law, B,, McGowan, B, and Munro, M. (1998) Managing Careers Work in Schools, NICEC 
Project Report, Cambridge: Careers Research and Advisory Centre. 

Audit Commission and Her Majesty’s Inspectorate (1993). Unfinished Business: Full-time Educational 
Courses for 16-19 Year Olds. London: HMSO. 

Azrin, N,H., Philip, R.A., Thienes-Hontos, P, and Besalel, V,A. (1980). Comparative evaluation of the Job 
Club program with welfare recipients. Journal of Vocational Behavior. 16, 133-145. 

Azrin, N.H,, Philip, R.A., Thienes-Hontos, P. and Besalel, V.A, (1981), Follow-up on welfare benefits 
received by Job Club clients. Journal of Vocational Behavior. 1 8 , 253-254, 

Barham, L., Hughes, D., and Morgan, S,, (2000) New Start - Paving the way for the Learning Gateway. 
Final evaluation of the personal adviser pilot projects, London: Department for Education and 
Employment. 

Blundell, R., Dias, M., Meghir, C, and Van Reenen, J. (2000) Evaluating the New Deal: what has been the 
effect of the Gateway? Institute of Fiscal Studies mimeo, 

Bosley, S., El-Sawad, A,, Hughes, D., Jackson, C, and Watts, A. G, (2001) Guidance and Individual 
Learning Accounts. Report. Derby: Centre for Guidance Studies, University of Derby, 

Brooks, R. (1998) Staying or leaving? A literature review of factors affecting the take-up of post- 16 
options. Slough: National Foundation for Educational Research. 

Bysshe, S. and Parson, D. (1999) Evaluation of Learning Direct. London: DfEE 

Campbell, D. P. (1965) The results of counselling: twenty-five years later, Philadelphia: Saunders. 

Cherry, N, (1974) Do Careers Officers give good advice British Journal and Guidance and Counselling, 
Vol 2, pp. 27-40. 

Connor, H,, and Dewson, S., with Tyers, C., Eccles, J., Regan, J,, and Aston, J., (2001) Social Class and 
Higher Education: Issues Affecting Decisions on Partcipation by Lower Social Class Groups: 
Sheffield. DFEE. 




20 



21 



Coopers and Ly brand (1995) National Evaluation of Skill Choice - Final Report, Employment Department 

Davies, V. and Irving, P. (2000) New Deal for Young People: Intensive Gateway Trailblazers. Research 
and Development Report ESR50. Sheffield: Employment Service. 

Eden, D. & Aviram, A. (1993) Self-efficacy training to speed reemployment: helping people to help 
themselves. Journal of Applied Psychology. Vol 78, pp. 352-360. 

Evans, J.H. and Burck, H.D. (1992). The effects of career education interventions on academic 
achievement: a meta-analysis. Journal of Counselling and Development, 71, 63-68. 

Davies, V. and Irving, P. (2000) New Deal for Young People: Intensive Gateway Trailblazers. Research 
and Development Report ESR50. Sheffield: Employment Service. 

Fretz, B.R. (1981) Evaluating the effectiveness of career interventions. Journal of Counseling Psychology, 
28: 77-90. 

Gardiner, K. (1997) Bridges from benefit to work: a review, York Publishing Services Ltd: York 

Harmon, C., and Walker, I., (2001) The Returns to Education: A review of Evidence, Issues and 
Deficiencies in the Literature. 

Hasluck, C. (2000a) The New Deal for Young People: two years on. Research and Development Report 
ESR41. Sheffield: Employment Service. 

Hasluck,C.(2000b) The New Deal for Lone Parent: A review of evaluation evidence. Sheffield: 
Employment Service. 

Hawthorn, R., and Watts, A.G., (1992) Careers Education and the Curriculum in Higher Education. 
Cambridge: Hobsons Publishing on behalf of CRAC. 

Heckman, J., Ichimura, H. and Todd, P. (1998) Matching As An Econometric Evaluation Estimator, 
Review of Economic Studies, 65, 261-294. 

Herr, E.L. (2001) The Costs/Benefits of Career development Interventions: A Practitioner's Perspective. 
Paper presented to Second International Symposium on Career Development and Public Policy, 
Vancouver, Canada. 

Higher Education Funding Council for England (1999). Widening Participation in Higher Education: 
Funding Decisions. Report 99. 

Higher Education Quality Council (HEQC), (1994), Guidance and Counselling in Higher Education. 
London : HEQC. 

Hirsh W., Jackson C., Kidd J. (2001) Straight Talking: Effective Career Discussions at Work. Cambridge: 
National Institute of Career Education and Counselling. 

Hughes, D., Bosley, S., Bowes, L. and Bysshe, S. (2002) The Economic Benefits of Guidance, Centre for 
Guidance Studies Report, University of Derby. 

Imbens, G. and Angrist, J. (1994) Identification and estimation of local average treatment effects. 
Econometrica, 62(2) 467-476. 




21 



22 



Jackson, C. Watts, A. G., Hughes, D., Bosley, S. and El-Sawad, A. (2001) Careers service work with 
adults: a survey. Occasional Paper, Derby: Centre for Guidance Studies, University of Derby. 

James, K. (2001) Prescriptions for Learning: evaluation report. Leicester: National Institute for Adult 
Continuing Education. 

Keller, K. Glauber, D, & Snyder, J. (1983) Beliefs-focused and skills-focused employment interventions 
for women. Journal of Employment Counselling. Vol 20. Pp. 163-168. 

Killeen, J (1996a) The Learning and Economic Outcomes of Guidance, (1996b) Evaluation in Watts, A, G, 
Law, B, Killeen, J, Kidd, J. M, and Hawthorn, R (1996) Rethinking Careers Education and 
Guidance Theory, Policy and Practice, London: 

Killeen, J., (1996b). Does guidance work? An evaluation of the intermediate outcomes of Gateways to 
Learning. London: Department for Education and Employment. 

Killeen, J. and Kidd, J. (1991) Learning outcomes of guidance: a review of recent research. Employment 
Department Research Paper 85. 

Killeen, J., Kidd, J.M., Hawthorn, R., Sampson, J. and White, M. (1994) A Review of Measures of the 
Learning Outcomes of Guidance. Cambridge: NICEC. 

Killeen, J., Sammons, S. and Watts, A.G. (1999) The effects of careers education and guidance on 
attainment and associated behaviour. Cambridge: NICEC/University of Hertfordshire/ ULIE. 

Killeen, J., Sammons, S. and Watts, A.G. (1999) The effects of careers education and guidance on 
attainment and associated behaviour. Cambridge: National Institute for Careers Education and 
Counselling. 

Killeen, J. and White M. (2000) The Impact of Careers Guidance on Adult Employed People, DfEE 
Research Report RR226, Sheffield: Department for Education and Employment. 

Killeen, J., White, M. and Watts, A.G. (1992). The Economic Value of Careers Guidance. London: Policy 
Studies Institute. 

Lapan, R.T., Gysbers, N.C. and Sun, Y. (1997). The impact of more fully implemented guidance programs 
on the school experiences of high school students: a statewide evaluation study. Journal of 
Counselling and Development, 75, 292-301. 

La Valle, I. and Finch, S. (1999) Pathways in Adult Learning Survey. London: DfEE 

Martinez, P. (1995) Factors affecting student retention. Mendip Report. Coventry: Further Education 
Development Agency. 

Martinez, P. and Munday, F. (1998) 9,000 voices: student persistence and drop-out in further education. 
LSDA Report. Coventry: Further Education Development Agency. 

Mayston, D. (2002a). Evaluating the Benefits of Career Guidance, Centre for Guidance Studies Report, 
University of Derby. 

Mayston, D. (2002b). Assessing the Benefits of Career Guidance, Centre for Guidance Studies Occasional 
Paper, University of Derby. 





McGivney, V. (1996) Staying or leaving the course: retention and non-completion of mature students in 
further and higher education. Leicester: National Institute for Adult Continuing Education. 

McLeman, P., and Smith, R, The Career Management Initiative at Buckinghamshire Chiltems University 
College, in Yorke, M., and Stephenson, J., (1998) Capability and Quality in Higher Education 
London: Kogan Page 

MORI (2001) Demand for Information, Advice and Guidance, The Guidance Council: Winchester 

MORI (1996) Evaluation of ESF Vocational Guidance and Counselling Schemes, Sheffield: DFEE. 

Morris, M., Nelson, J., Rickinson, M., and Stoney, S.M., with Benefield A. (1999) A literature review of 
young people* s attitudes towards education, employment and training. London: Department for 
Education and Employment. 

Morris, M. and Schagen, I. with Weston, P. (1996). The Outcomes of Careers Education and Guidance for 
Young People: Final Report. National Foundation for Educational Research (unpublished). 

Reported by Saunders, L. Stoney, S. and Weston, P. The impact of the work-related curriculum on 
14- to 16-year olds. Journal of Education and Work, 10(2) 151-167. 

Mountain, E. (2001) What Quality Outcomes for career Development Across All Ages do Policy Makers 
and Practitioners Jointly Endorse Now? What Might Be/Must Be Added? Paper presented to Second 
International Symposium on Career Development and Public Policy, Vancouver, Canada 

Myers, R.A. (1986). Research on educational and vocational counseling. In Bergin, A.E. and Garfield, 

S.L. (Eds.), Handbook of Psychotherapy and Behavior Change (3^ Edition), New York: Wiley. 

Oliver, L (1979) Outcome measurement in career counseling. Journal of Counseling Psychology, 26(3): 
217-226. 

Oliver, L.W. and Spokane, A.R. (1988) Career intervention outcome: what contributes to client gain? 
Journal of Counseling Psychology, 35(4): 447-462. 

Park, A. (1994) Individual Commitment to Learning: Individuals* Attitudes. Report on the quantitative 
survey. Employment Department, Research Series No. 32 

Rivis, V., Assuring the Quality of Guidance and Learner Support in Higher Education in Wisker, G., and 
Brown, S., (1996) Enabling Student Learning: Systems and Strategies. London : Kogan Page. 

Rosenbaum, P.R. and Rubin, D.B. (1983) The central role of the propensity score in observational studies 
for causal effects. Biometrika, 70, 41-55. 

Saunders, L. Stoney, S and Weston P. (1997) The impact of the work -related curriculum on 14 to 16 year 
olds. Journal of Education and Work 10(2) pp 151 et seq. 

Savickas, M.L. (2001) Quality Outcomes for Career Development: The Perspectives of Policy and 

Practice. Paper presented to Second International Symposium on Career Development and Public 
Policy, Vancouver, Canada. 

Sillitoe, K. and Meltzer, H. (1985) The West Indian school-leaver W oh 1 and 2 London, HMSO. 




23 



24 



Sims, D., Nelson, J., Golden, S. and Spielhofer, T. (2001 j Young people's experiences of the Learning 
Gateway. Research Report RR277. London: Department for Education and Skills. 

Skills Task Force (2000) Skills for All: Research Report from the National Skills Task Force. DFEE, 
Sheffield. 

Spokane, A.R. and Oliver, L.W. (1983) The outcomes of vocational intervention, in Walsh, W.B. and 
Osipow, S. H. (Eds) Handbook of Vocational Psychology, Volume 2. Hillsdale, NJ: Lawrence 
Erlbaum. 

Strategies for Widening Participation in Higher Education: A guide to good practice. Higher Education 
Funding Council for England (2001) Report 01/36 June 2001 

SWA Consulting (1999) Evaluation of early Individual Learning Account development activity. Research 
Report 123. London: DfEE. 

The National Committee of Inquiry into Higher Education (NCME). Higher Education in a Learning 
Society 

Tremlett, N., Park, A. and Dundon-Smith, D. (1995) Individual commitment to learning. Further findings 
from the individual surveys. Research Series 54. Social and Community Planning Research. 
Sheffield: Employment Department. 

Tremlett, N., Thomas, A. and Taylor, S. (1995) Individual commitment to learning: providers' attitudes. 
Research Series 47. Sheffield: Employment Department. 

Unit for the Development of Adult Continuing Education (1986) The Challenge of Change, Leicester: 
UDACE. 

Van Reenen, J., (2001) No more skivvy schemes? Active labour market policies and the British New Deal 
for the young unemployed in context. WPOl/09. London: The Institute for Fiscal Studies. 

Verma, G.K. and Darby, D.S. (1987). Race, Training and Employment. Lewes: Falmer. 

Watts, A.G. (1999) The Economic and Social Benefits of Guidance, Educational and Vocational 

Guidance: Bulletin, 63/1999. International Association for Educational and Vocational Guidance 

Watts, A.G. and Herr, E.L. (1976). Career(s) education in Britain and the USA: contrasts and common 
problems. British Journal of Guidance and Counselling, 4(2), 129-142. 

Watts. A.G., Law. B., Killeen. J., Kidd. J.M., & Hawthorn.R. (1996) Rethinking Careers Education & 
Guidance: theory, policy and practice. London: Routledge. 

Whiston, S.C., Sexton, T.L. and Lasoff, D.L. (1998). Career-Intervention Outcome: A Replication and 
Extension of Oliver and Spokane. Journal of Counseling Psychology, 45(2) 150-165. 

White, M. (1992) The potential of guidance: ideas from economics, in Killeen J. et al, op cit. 

Williamson, E.G. and Bordin, E.S. (1940) Evaluating counselling by means of a control group experiment. 
School and Society, Vol 52, pp. 434-440. 





24 



Witherspoon S (1995) Careers advice and the careers service: the experiences of young people ED 
Research Series. Youth Cohort Report 33 Employment Department. 

Winterbotham, M., and Adams, L., with Hasluck, C., (2001) Evaluation of New Deal for Long Term 
Unemployed People Enhanced National Programme, Report ESR82. Sheffield: Employment 
Service. 

Yorke, M. Ozga, J. and Sukhnanden, L. (1997) Undergraduate non-completion in higher education in 
England. Higher Education Funding Council for England. 




25 




U.S. Department of Education 

Office of Educational Research and Improvement (OERI) 
National Library of Education (NLE) 
Educational Resources Information Center (ERIC) 







E^ucfltlonal Resources loloriiidion Center 



NOTICE 



Reproduction Basis 



X 



This document is covered by a signed "Reproduction Release (Blanket)" 
form (on file within the ERIC system), encompassing all or classes of 
documents from its source organization and, therefore, does not require a 
"Specific Document" Release form. 



This document is Federally-funded, or carries its own permission to 
reproduce, or is otherwise in the public domain and, therefore, may be 
reproduced by ERIC without a signed Reproduction Release form (either 
"Specific Document" or "Blanket"). 



o EFF-089 (1/2003) 

ERIC 



